Repository navigation
modify emit() of TopK to emit on batch_size rather than batch_size-1 - #10030
Conversation
alamb
left a comment
There was a problem hiding this comment.
makes sense to me -- thank you @JasonLi-cn
batch_size rather than batch_size-1
|
(BTW @JasonLi-cn I am not sure what is inspiring all your work recently to optimize DataFusion but it is much appreciated 🙏 ) |
|
Thanks @JasonLi-cn |
I have been using DataFusion and arrow-rs since 2021, but most of my work has been on developing TableScan or upper-level applications, as well as some optimizations for specific scenarios. My current work is mainly focused on performance optimization (which is also my interest) and feature enhancement, and I think it is necessary to contribute the improvements directly to the community, which is beneficial for the healthy development of both DataFusion and our own projects. |
100%
I think this describes everyone here (including myself!) I also learn a lot from interacting with other developers and working with the communnity |
Which issue does this PR close?
Closes #.
Rationale for this change
Simply modify the logic of "break into record batches as needed". This modification avoid generating an empty RecordBatch where the batch.num_rows() eq batch_size.
https://github.com/apache/arrow-datafusion/blob/03d8ba1f0d94bac6bb8bb33e95f00f9f6fb5275a/datafusion/physical-plan/src/topk/mod.rs#L210-L219
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?